Point clouds are collections of points in a three dimensional system that describe a 3D scene or a 3D object. Each point in the point cloud represents XYZ coordinates within a 3D coordinate system. Typically, the points within a point cloud represent the exterior surfaces of a 3D object. Point clouds are typically generated by a 3D scanning technology such as, for example, light detection and ranging (LIDAR) systems. However, point clouds can also be generated by other 3D scanning technologies including, for example, depth sensors, structured-light depth cameras (which can detect depth and 3D shapes based on projected light patterns), laser triangulation sensors, or through photogrammetric techniques (e.g. constructing a 3D point cloud based on measurements of a scene or object from multiple angles).
In conventional 3D scanning and registration systems, 3D scanner technologies are sometimes paired with two dimensional (2D) electro-optical systems (e.g. two dimensional camera sensors). The combination of 3D scanner technologies with 2D electro-optical systems provides for synergistic uses with many applications such as, for example, surveillance, robotics, video games, environmental modeling, aerospace flight and/or proximity detection. The 2D electro-optical systems and 3D scanner systems also have their respective characteristics and advantages which can improve a 3D point cloud generated by the 3D scanner. For example, 2D electro-optical systems typically have a higher resolution than a point cloud generated by a 3D scanner (such as, for example, a LIDAR sensor). Further, the 3D upsampling of 3D scanner data using 2D electro-optical information can provide for more detailed point clouds with higher point cloud density (e.g. a higher point cloud resolution).
In conventional systems, upsampling of 3D point clouds with 2D electro-optical data from 2D electro-optical sensors includes using multi-resolution depth maps based on visibility analysis to perform more accurate upsampling while removing outlier points within the 3D point cloud. This conventional upsampling system performs best with simple 3D structures like tall buildings, but often fails to accurately upsample point clouds with more complex and more irregularly shaped objects such as, for example, foliage or trees. Other conventional systems include using high-quality depth map upsampling methods to obtain cleaner edges between objects. High-quality depth map upsampling provides for obtaining edge discontinuities based on heuristic weighting factors that use only the global structure of objects. Other conventional systems employ high dynamic range cameras in an electro-optical/3D scanner upsampling fusion, which is helpful in environments with poor lighting (e.g., using high dynamic range cameras to compensate for light blowouts in bright environments and black-crush in darker environments), while employing a time-consuming Markov Random Field framework for sharper shape reconstruction from illumination. Yet other conventional systems include a depth point cloud upsampling method that backfills sparse point cloud areas using electro-optical intensity information by iteratively fusing electro-optical pixel information with 3D scanner data by sliding backfilling windows at different scales.
Conventional electro-optical/3D scanner upsampling systems all generate upsampled 3D point clouds for better visualization characteristics. However, the conventional electro-optical/3D scanner upsampling methods often generate new 3D points in the point cloud that reduce overall 3D registration accuracy by presenting greater chances of introducing or generating outlier points. Further, conventional electro-optical/3D scanner upsampling systems are typically computationally expensive and not suited for real-time operations such as, for example, space rendezvous, space docking systems, or proximity maneuvering.